Information processing apparatus, shooting system, method, and non-transitory computer-readable storage medium

ABSTRACT

There is provided with an information processing apparatus. A storage unit stores pieces of image quality information respectively for a plurality of images that show a subject. A determination unit determines whether image qualities of the images are favorable based on the pieces of image quality information of the images. A composition unit generates a composite image using images that have been determined to have favorable image qualities by the determination unit. A notification unit, in a case where there is a missing pixel area in the composite image, provides a notification about a re-shooting method for the subject corresponding to the missing pixel area.

BACKGROUND OF THE INVENTION CROSS REFERENCE TO PRIORITY APPLICATION

This application claims the benefit of Japanese Patent Application No.2021-208441, filed Dec. 22, 2021, which is hereby incorporated byreference herein in its entirety.

Field of the Invention

The present invention relates to an information processing apparatus, ashooting system, a method, and a non-transitory computer-readablestorage medium.

Description of the Related Art

An inspector of a structure, such as a bridge and a tunnel, checks thepositions and extents of deformations (cracks, water leakage, and soforth) on the surfaces of the structure, and records the inspectionresult. In general, a civil engineer visually checks whether there aredeformations on the surfaces of the structure, and records thepositions, sizes, and the like of the deformations on, for example, ahand-held notebook. However, in the recent years, there has been aproblem of a labor shortage in civil engineers who inspect structuresdue to, for example, aging of civil engineers. To address this problem,“image-based inspection” has been performed whereby the surfaces of astructure are shot with high definition using an image capturingapparatus (camera), and deformations are checked and recorded based onshot images using an image analysis technique.

In image-based inspection, an interchangeable-lens, high-definitionimage capturing apparatus (camera) that supports a large number ofpixels is mounted on, for example, an automatic tripod head or a drone,and the surfaces of a structure to be inspected are exhaustively shotusing the image capturing apparatus. In a case where the surfaces of astructure, such as a bridge and a tunnel, are shot with a resolutionthat allows detection of deformations (cracks), several tens to severalhundreds of shot images are obtained. However, as each shot image onlyshows a small area of the structure, an inspector cannot understandwhich area of the structure has been shot from each shot image. For thisreason, respective shot images are stitched (composited) to generate alarge-scale stitched image (composite image) that shows the surfaces ofthe structure in a wider range. In this way, by performing shooting insuch a manner that respective shooting ranges overlap one another at thetime of shooting, shot images can be composited together.

A technique has been disclosed that determines an abnormality, such asmissing data in shot images that have been shot by a shooting apparatusmounted on an unmanned flight vehicle, and specifies images that need tobe re-shot in accordance with whether there is an abnormality.Specifically, images to be re-shot are specified by detecting a dataabnormality, such as missing image data, at the time of communication ofshot images (Japanese Patent No. 6619761).

SUMMARY OF THE INVENTION

The present invention in its one aspect provides an informationprocessing apparatus comprising a storage unit configured to storepieces of image quality information respectively for a plurality ofimages that show a subject, a determination unit configured to determinewhether image qualities of the images are favorable based on the piecesof image quality information of the images, a composition unitconfigured to generate a composite image using images that have beendetermined to have favorable image qualities by the determination unit,and a notification unit configured to, in a case where there is amissing pixel area in the composite image, provide a notification abouta re-shooting method for the subject corresponding to the missing pixelarea.

The present invention in its one aspect provides an a shooting systemcomprising a mobile object including shooting unit configured to shoot asubject, and an information processing apparatus that processes imagesshot by the shooting unit, wherein the information processing apparatusincludes a storage unit configured to store pieces of image qualityinformation respectively for a plurality of images that show thesubject, a determination unit configured to determine whether imagequalities of the images are favorable based on the pieces of imagequality information of the images, a composition unit configured togenerate a composite image using images that have been determined tohave favorable image qualities by the determination unit, and anotification unit configured to, in a case where there is a missingpixel area in the composite image, provide a notification about are-shooting method for the subject corresponding to the missing pixelarea.

The present invention in its one aspect provides an a shooting systemcomprising a tripod head, a shooting unit configured to shoot a subject,the shooting unit being mounted on the tripod head, and an informationprocessing apparatus that processes images shot by the shooting unit,wherein the information processing apparatus includes a storage unitconfigured to store pieces of image quality information respectively fora plurality of images that show the subject, a determination configuredto determine whether image qualities of the images are favorable basedon the pieces of image quality information of the images, a compositionunit configured to generate a composite image using images that havebeen determined to have favorable image qualities by the determinationunit, and a notification unit configured to, in a case where there is amissing pixel area in the composite image, provides a notification abouta re-shooting method for the subject corresponding to the missing pixelarea.

The present invention in its one aspect provides a method comprisingstoring pieces of image quality information respectively for a pluralityof images that show a subject, determining whether image qualities ofthe images are favorable based on the pieces of image qualityinformation of the images, generating a composite image using imagesthat have been determined to have favorable image qualities by thedetermining, and in a case where there is a missing pixel area in thecomposite image, providing a notification about a re-shooting method forthe subject corresponding to the missing pixel area.

The present invention in its one aspect provides a non-transitorycomputer-readable storage medium storing a program that, when executedby a computer, causes the computer to perform a method of an informationprocessing apparatus comprising storing pieces of image qualityinformation respectively for a plurality of images that show a subject,determining whether image qualities of the images are favorable based onthe pieces of image quality information of the images, generating acomposite image using images that have been determined to have favorableimage qualities by the determining, and in a case where there is amissing pixel area in the composite image, providing a notificationabout a re-shooting method for the subject corresponding to the missingpixel area.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram showing an exemplary configuration of functions ofan information processing apparatus.

FIG. 1B is a diagram showing examples of determination informationobtained by determining the image quality of shot image data.

FIG. 2 is a diagram showing an example of a composite image generated bythe information processing apparatus.

FIG. 3 is a diagram showing an example of presentation of re-shootingmethods to be used by the information processing apparatus.

FIG. 4 is a flowchart illustrating a flow of processing executed by theinformation processing apparatus.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention. Multiple features aredescribed in the embodiments, but limitation is not made an inventionthat requires all such features, and multiple such features may becombined as appropriate. Furthermore, in the attached drawings, the samereference numerals are given to the same or similar configurations, andredundant description thereof is omitted.

According to the present invention, a subject that needs to be re-shotcan be efficiently shot.

(First Embodiment)

An information processing apparatus stores respective pieces of imagequality information of a plurality of images that show a subject, anddetermines whether the image qualities of the images are favorable basedon the pieces of image quality information of the images. Theinformation processing apparatus generates a composite image with use ofimages that have been determined to have favorable image qualities, andin a case where there is a missing pixel area in the composite image,provides a notification about a method of re-shooting a subjectcorresponding to the missing pixel area. Note that although theinformation processing apparatus is mounted on a mobile object (drone)or a tripod head, it may be mounted on any apparatus on which theinformation processing apparatus can be mounted. Furthermore, theinformation processing apparatus may not be mounted on a mobile object(drone) or a tripod head, and an information processing apparatus thatis separately prepared may control a mobile object (drone) or a tripodhead via, for example, wireless communication. Moreover, data may beexchanged between a mobile object (drone) or a tripod head and theinformation processing apparatus via a removable storage medium. In thiscase, a shooting system that includes the information processingapparatus and the mobile object (drone) is configured. Here, imagequality information denotes a plurality of elements with which the imagequality of an image obtained by shooting a subject is evaluated (e.g., ashooting resolution, focus, and blurring). The image quality denotes thequality of an image obtained by shooting a subject. Whether the imagequality is favorable is determined based on whether at least one of theplurality of elements of the image quality information is favorable. Acomposite image denotes one image obtained by compositing a plurality ofimages obtained by shooting a subject.

The following describes the reason why a technique to assist aninspector in determining the image quality and making a judgment onre-shooting is necessary. For example, in inspection of an extremelythin crack with a width of 0.2 mm on the surface of a structure, theextremely thin crack may not be shown in images unless the shootingsettings of an image capturing apparatus (camera) are appropriatelyconfigured. Among the shooting settings, especially a shootingresolution (an actual dimension corresponding to one pixel in an image),focus (focal point), and blurring are important in ensuring the qualityof the image (hereinafter, image quality). When the inspector inspects acrack with a width of 0.2 mm or more, it is sufficient that the shootingresolution be higher than 0.5 mm/pixel. When the inspector inspects acrack with a width of 0.05 mm or more, it is sufficient that theshooting resolution be higher than 0.3 mm/pixel. It is possible to shoota crack thinner than the shooting resolution; however, in shooting of amuch thinner crack, it is necessary to shoot the crack with a highershooting resolution. For example, in a bridge inspection under thedomestic jurisdiction, cracks with a width of 0.05 mm or more are thetargets of inspection, and it is recommended to perform shooting with ashooting resolution higher than 0.3 mm/pixel (the Ministry of Land,Infrastructure, Transport and Tourism (March 2021, Manual for Deliveryof Three-Dimensional Deliverable Using Inspection Assistance Technique(Image Measurement Technique (Draft)).

Focus adjustment exerts a large influence on how cracks are shown inshot images. For example, in images obtained by shooting cracks in anout-of-focus state, extremely thin cracks are not shown due to defocus.In image-based inspection, as the entirety of shot images shows thesurface of a structure (hereinafter, an inspection surface), it isnecessary to configure settings related to shooting of an imagecapturing apparatus so as to bring the entirety of the shot images intofocus. That is to say, it is necessary to set a diaphragm (f-number) sothat the depth of field (the range that is in focus) includes theclosest point through to the farthest point of the inspection surface.The larger the f-number, the deeper the depth of field, and the widerthe range that is in focus. However, increasing the f-number will reducethe shutter speed, thereby making a subject (cracks) easily blurred.

Also, a drastic increase in the f-number causes defocus called“small-aperture defocus”, and exerts an influence on how cracks areshown. Furthermore, shaking of an image capturing apparatus (camera) atthe time of shooting of cracks exerts an influence on how cracks areshown. In blurry images, the appearance of cracks are such that thecracks are not easily distinguishable from stains on the surface of astructure. Although increasing the ISO film speed can increase theshutter speed of the image capturing apparatus (camera), setting the ISOfilm speed too high will cause ISO noise. As the ISO film speed exertsan influence on how cracks are shown in shot images, it is necessary toadjust the shooting conditions so that shooting can be performed withoutblur while avoiding an increase in the value of the ISO film speedwhenever possible.

As described above, the matters to be kept in mind during shooting atthe time of inspection of a structure are clear; however, as optimumsetting values vary depending on the shooting condition, they cannot bedetermined in advance. For example, the focal length that realizes anecessary shooting resolution varies depending on the distance from theimage capturing apparatus (camera) to the target of shooting (a crack).As a necessary depth of field varies depending on the direction of thesurface to be shot (the surface with a crack), an optimum f-numbervaries. As the brightness of the surrounding of the target of shooting(a crack) also varies depending on the weather, the shutter speednecessary for shooting images without blur varies as well. For example,in a case where the surface of a structure is shot using an imagecapturing apparatus (camera) mounted on a drone, the distance betweenthe image capturing apparatus (camera) and the target of shooting, aswell as the direction of the surface to be shot, may change immediatelybefore shooting depending on the wind condition at the time of shooting.Furthermore, an optimum value of a shutter speed for shooting withoutblur changes due to a sudden gust of wind at the time of shooting.

In view of the above-described conditions, the image qualities of shotimages are determined after the surface of a structure has been shot,and if there is a shot image that is inappropriate for inspection by aninspector, the location to be re-shot and the method for improving theimage qualities of shot images are specified, and re-shooting isperformed. Here, in terms of cost reduction, the determination of theimage qualities of shot images and the judgment on whether re-shootingis necessary are made immediately after shooting at the site ofshooting. For example, in a case where a structure to be inspected isfar from the office of the inspector, and in a case where it isnecessary to make arrangements to, for example, rent shooting equipment,re-shooting of the target of inspection is difficult because it incurs alarge amount of effort. In general, the determination of the imagequalities of shot images and the judgment on whether re-shooting isnecessary are made based on the experience and know-how of theinspector; however, it is difficult to make these determination andjudgment appropriately in a short amount of time. In view of this, thereis a need for a technique to assist the inspector in determining theimage qualities and making a judgment on re-shooting.

FIG. 1A is a diagram showing an exemplary configuration of functions ofan information processing apparatus. An information processing apparatus100 includes a shooting unit 101, a storage unit 102, a determinationunit 103, a composition unit 104, and a notification unit 105.

The shooting unit 101 is an image capturing apparatus that shoots asubject, and includes a camera mounted on, for example, a mobile object(e.g., a drone), a tripod head, and the like. The shooting unit 101captures images of the surfaces of a structure to be inspected (referredto as inspection target surfaces), and generates shot image data. “Shotimage data” includes image data of an inspection target, model names ofthe image capturing apparatus and a lens, such camera parameters as thefocal length, shutter speed, and ISO film speed at the time of shooting,information of a shooting distance from the shooting unit 101 to thesubject, and pieces of in-focus degree information of a shot image(defocus values). The shooting unit 101 obtains information of ashooting distance from the shooting unit 101 to the subject with use of,for example, known means, such as a distance measurement apparatusincluded in the shooting unit 101.

The shooting unit 101 includes, for example, two photoelectricconversion units in which a sensor included in the shooting unit 101performs photoelectric conversion on a per-pixel basis. The shootingunit 101 calculates pieces of in-focus degree information (defocusvalues) of a shot image based on the phase difference between two imagesthat are respectively recorded in the photoelectric conversion units. Apiece of in-focus degree information is represented by, for example, avalue of 0.0 or more for each pixel. In a case where a piece of in-focusdegree information indicates 0.0, it means that there is no phasedifference between two images in that pixel, and that pixel is in focus.In a case where a piece of in-focus degree information indicates 0.0 ormore, it means that there is a phase difference corresponding to anumerical value between two images in that pixel, and that pixel is outof focus.

The mobile object (drone) or tripod head on which the shooting unit 101is mounted may be controlled by means of a user operation or anautomatic operation. No matter which one of the user operation and theautomatic operation controls the shooting unit 101, the shooting unit101 obtains a plurality of shot images by exhaustively shooting theinspection target surfaces while changing a shooting range with respectto the subject. In order to cause the composition unit 104 to compositethe shot images, the shooting unit 101 shoots images in such a mannerthat the shooting ranges of respective inspection target surfacesoverlap (overlie) one another.

For example, the shooting unit 101 may perform shooting in such a mannerthat respective shooting ranges overlap (overlie) one another byrepeating still image shooting. Alternatively, the shooting unit 101 mayobtain shot image data in which respective shooting ranges overlap oneanother by extracting (capturing) still images from moving images afterthe moving images have been shot. Here, the timing of shooting of stillimages and the timing of extraction of still images from moving imagesmay be no particular timings based on a user setting, or may be timingsautomatically set by the shooting unit 101. The shooting unit 101transmits a plurality of pieces of shot image data that have beenobtained to the storage unit 102.

The storage unit 102 is a storage apparatus that stores various types ofdata inside the information processing apparatus 100, and includes, forexample, an HDD, an SSD, a RAM, a ROM, and the like. The storage unit102 receives the pieces of shot image data from the shooting unit 101,and stores the pieces of shot image data into various types of storagemediums. Under an instruction from a CPU (not shown) included in theinformation processing apparatus 100, the storage unit 102 transmits thepieces of shot image data to the determination unit 103. The storageunit 102 receives determination information obtained as a result of thedetermination unit 103 determining whether the image qualities of thepieces of shot image data are favorable (equivalent to a determinationresult). Under an instruction from the CPU (not shown), the storage unit102 transmits only the pieces of shot image data with favorable imagequalities to the composition unit 104 based on the determinationinformation.

The determination unit 103 receives pieces of shot image data from thestorage unit 102, and determines the image qualities of the pieces ofshot image data. The determination unit 103 transmits determinationinformation obtained by determining the image qualities of the pieces ofshot image data to the storage unit 102 and the notification unit 105.The determination unit 103 calculates a shooting resolution, which isincluded in image quality information, with use of the followingexpression 1 based on a shooting distance from the shooting unit 101 toa subject (an inspection target surface), the camera’s sensor size, thefocal length, and the number of pixels. Image quality informationincludes, for example, a shooting resolution, focus, and blurring.

$\text{Shooting}\mspace{6mu}\text{resolution}\mspace{6mu}\text{=}\frac{\text{Shooting}\mspace{6mu}\text{distance}\mspace{6mu} \times \mspace{6mu}\text{Sensor}\mspace{6mu}\text{size}}{\text{Focal}\mspace{6mu}\text{length}\mspace{6mu} \times \mspace{6mu}\text{Number}\mspace{6mu}\text{of}\mspace{6mu}\text{pixels}}$

The shooting distance and the focal length are information recorded inshot image data. The sensor size and the number of pixels are numericalvalues that are unique to each of the models of image capturingapparatuses (cameras). Therefore, the determination unit 103 obtains theunique numerical values with reference to, for example, a databasestored in the storage unit 102 or the like based on model information ofthe image capturing apparatus (camera) recorded in shot image data. Thedetermination unit 103 determines the image quality of shot image databy comparing the shooting resolution calculated based on expression 1with a base shooting resolution that has been recorded in advance in thestorage unit 102 or the like. That is to say, the determination unit 103determines that the image quality of shot image data is favorable in acase where the numerical value of the shooting resolution is equal to orlarger than the numerical value of the base shooting resolution. On theother hand, the determination unit 103 determines that the image qualityof shot image data is not favorable in a case where the numerical valueof the shooting resolution is smaller than the numerical value of thebase shooting resolution.

Next, a description is given of an example in which the determinationunit 103 determines whether the image quality of shot image data isfavorable with use of “focus” included in image quality information. Thedetermination unit 103 determines whether the image quality is favorablebased on pieces of in-focus degree information recorded in shot imagedata. Shot image data includes, for example, values of pieces ofin-focus degree information of respective pixels in a shot image. Asdescribed earlier, in a case where the numerical value of a piece ofin-focus degree information of each pixel is 0.0, it means that thepixel is in focus. In a case where the numerical value is a numericalvalue larger than 0.0, it means that the pixel is out of focus inaccordance with that numerical value.

The determination unit 103 determines whether focus is favorable withrespect to each shot image based on the extent to which the entirety ofthe shot image includes pixels with a predetermined numerical value orless. For example, in a case where pixels with pieces of in-focus degreeinformation having a numerical value of 1.0 or less account for at least50% of the entirety of the shot image, the determination unit 103determines that focus of the shot image is favorable. On the other hand,in a case where pixels with pieces of in-focus degree information havinga numerical value of 1.0 or less account for less than 50% of theentirety of the shot image, the determination unit 103 determines thatfocus of the shot image is not favorable.

Furthermore, a description is given of an example in which thedetermination unit 103 determines whether the image quality of shotimage data is favorable with use of “blurring” included in image qualityinformation. The determination unit 103 determines whether blurring isincluded by performing frequency analysis with respect to image datarecorded in shot image data. In a case where shot image data includesblurring, cracks and the like are shown in a wide range in a shot image,and thus the shot image data has a low frequency. In view of this, thedetermination unit 103 calculates a spatial frequency for each area inthe shot image with use of, for example, a known method, such as aFourier transform. The determination unit 103 determines whether theshot image data includes blurring based on an average value obtained byaveraging the frequency components that have been calculated forrespective areas in the shot image. That is to say, in a case where theaverage value indicates a high frequency, the determination unit 103determines that the shot image data does not include blurring. On theother hand, in a case where the average value indicates a low frequency,the determination unit 103 determines that the shot image data includesblurring.

The determination unit 103 determines the image quality of shot imagedata from the storage unit 102, generates “determination information”indicating whether the image quality of the shot image data isfavorable, and transmits the determination information to the storageunit 102 and the notification unit 105. Here, in a case where thedetermination unit 103 determines one element included in image qualityinformation from the shot image data (one of the shooting resolution,focus, and blurring), a determination result related to one element isused as the determination information. In a case where the determinationunit 103 determines a plurality of elements included in image qualityinformation from the shot image data, a determination result based onthe combination of the plurality of elements is used as thedetermination information.

FIG. 1B is a diagram showing examples of determination informationobtained as a result of the determination unit 103 determining the imagequality of shot image data (a determination result). In a table ofdetermination information 110, ○ indicates that the image quality isfavorable, and × indicates that the image quality is not favorable. Thedetermination information 110 includes numbers indicating thedetermination order, the shooting resolution, focus, blurring, and thedetermination about the image quality. For example, in a case where thedetermination unit 103 determines that all elements of image qualityinformation (the shooting resolution, focus, and blurring) arefavorable, it generates determination information (a determinationresult) indicating that the image quality of shot image data isfavorable. Specifically, in a case where the shooting resolution is ○,focus is ○, and blurring is ○ with regard to “number = 1”, thedetermination unit 103 determines that the image quality is ○.

On the other hand, in a case where the determination unit 103 determinesthat one of the elements of image quality information (e.g., blurring)is not favorable, it generates determination information 110 indicatingthat the image quality of shot image data is not favorable.Specifically, in a case where the shooting resolution is ○, focus is ○,and blurring is × with regard to “number = 2”, the determination unit103 determines that the image quality is ×. Alternatively, for example,in a case where the determination unit 103 determines that one of theelements of image quality information (e.g., the shooting resolution) isfavorable, it may generate determination information indicating that theimage quality of shot image data is favorable. Specifically, in a casewhere the shooting resolution is ○, focus is ×, and blurring is × withregard to “number = 3”, the determination unit 103 determines that theimage quality is ○. Note that as shown in FIG. 1B, determinationinformation 110 may include a determination result about respectiveelements (the shooting resolution, focus, and blurring) for evaluatingthe image quality of shot image data.

The composition unit 104 receives pieces of shot image data from thestorage unit 102, and generates composite image data by compositing thepieces of shot image data using a known method. For example, in a casewhere pieces of shot image data that have been shot using a fixed tripodhead are to be composited, the composition unit 104 composites togetherthe pieces of shot image data using a method of panoramic composition.Also, in a case where pieces of shot image data that have been shotusing a mobile object (drone) are to be composited, the composition unit104 composites together the pieces of shot image data using a method ofthree-dimensional reconfiguration. In a case where the foregoing methodsare used, the composition unit 104 is not always capable of compositingall pieces of shot image data. For example, in a case where thecomposition unit 104 generates a composite image using pieces of shotimage data that include a few areas that overlap with (overlie) otherpieces of shot image data, it may not be capable of generating acomposite image with no missing pixel. A missing pixel means a pixel forwhich no pixel value has been recorded.

FIG. 2 is a diagram showing an example of a composite image generated bythe information processing apparatus. In FIG. 2 , the composite image201 includes an area 202, an area 203, and an area 204. The area 202 isan area in which a total of eight pixels are missing, and is “a missingsection A attributed to a shortage of overlap (overlying images)”. Thereare cases where the composition unit 104 generates the composite image201 including missing pixels as indicated by the area 202 of thecomposite image 201.

Also, the pieces of shot image data that are used to generate thecomposite image 201 do not represent all of the pieces of shot imagedata that have been shot by the shooting unit 101, but consist only ofpieces of shot image data for which the image qualities have beendetermined to be favorable by the determination unit 103. Therefore, thesection (pixels) for which the image qualities have not been determinedto be favorable in pieces of shot image data is not used to generate thecomposite image 201, and is thus displayed as the area 203 in thecomposite image 201. The area 203 is an area in which a total of 13pixels are missing, and is “a missing section B attributed to poor imagequality”.

Furthermore, in a case where pieces of shot image data include an unshotsection because the shooting unit 101 failed to shoot a subject(inspection target surfaces) due to a certain reason, this section isdisplayed as the area 204 in the composite image 201. Examples of thecertain reason include a case where the shooting unit 101 mounted on amobile object (drone) that is moving at high speed cannot shoot adesired subject area, and a case where a shutter cannot be released dueto, for example, a poor connection at a contact point between the imagecapturing apparatus and the lens in the shooting unit 101. The area 204is an area in which one pixel is missing, and is “a missing section Cattributed to a failure in shooting”.

As described above, after generating the composite image 201, thecomposition unit 104 specifies the areas 202 to 204 in the compositeimage 201 for which no pixel value has been recorded (pixels aremissing). Next, the composition unit 104 estimates the reasons formissing pixels in the area 204 (missing section C), the area 203(missing section B), and the area 202 (missing section A), in statedorder. Note that the order of estimation of the reasons for missingpixels is exemplary, and no limitation is intended by this.

For example, the composition unit 104 specifies “used pieces of shotimage data” which compose the surroundings of the area 203 and the area204 and which were used to generate the composite image 201, andspecifies used pieces of shot image data with shooting orders that areadjacent to shooting orders of used pieces of shot image data. Forexample, it is expected that there is no unallocated number (missingdata) in used pieces of shot image data with shooting orders that areadjacent to shooting orders of used pieces of shot image data thatcompose the surrounding of the area 204 (the missing section Cattributed to a failure in shooting). For example, the composition unit104 can determine whether there is an unallocated number (missing data)in pieces of shot image data based on whether there are unique numbers(e.g., serial numbers) that have been respectively allocated to usedpieces of shot image data. Therefore, in a case where the compositionunit 104 determines that there is no unallocated number (missing data)in used pieces of shot image data located in the surrounding of the area204, it estimates that the reason for missing pixels in the area 204(missing section C) is a “shortage of images”. Here, the “shortage ofimages” includes the reasons for missing pixels attributed to “a failurein shooting” and “a shortage of overlap”. Note that the composition unit104 can determine whether the reason for missing pixels in the area 204(missing section C) is “a failure in shooting” or “a shortage ofoverlap” as described later.

On the other hand, in a case where there is an unallocated number(missing data) in used pieces of shot image data with shooting ordersthat are adjacent to shooting orders of used pieces of shot image data,it means that there is shot image data that the composition unit 104 hasnot received from the storage unit 102. That is to say, it is expectedthat there is an unallocated number (missing data) in used pieces ofshot image data with shooting orders that are adjacent to shootingorders of used pieces of shot image data that compose the surrounding ofthe area 203 (the missing section B attributed to poor image quality).Therefore, in a case where the composition unit 104 determines thatthere is an unallocated number (missing data) in used pieces of shotimage data located in the surrounding of the area 203, it estimates thatthe reason for missing pixels in the area 203 is “poor image quality”.

As another estimation method, the composition unit 104 may specify, fromamong pieces of shot image data received from the storage unit 102,pieces of shot image data that were not used to generate the compositeimage 201 (referred to as unused pieces of shot image data). Thecomposition unit 104 specifies pieces of shot image data which haveshooting times close to shooting times of the unused pieces of shotimage data (e.g., the difference between the former shooting times andthe latter shooting times falls within a predetermined range), and whichwere used to generate the composite image 201 (referred to as usedpieces of shot image data). The composition unit 104 specifies a portionof the composite image 201 that the used pieces of shot image datacompose (i.e., the positions of the used pieces of shot image data inthe composite image 201). For example, it is expected that used piecesof shot image data that compose the surrounding of the area 203 (themissing section B attributed to poor image quality) in the compositeimage 201 have shooting times close to shooting times of the unusedpieces of shot image data. In view of this, in a case where a missingpixel area is included in the surrounding of the used pieces of shotimage data specified from the composite image 201, the composition unit104 estimates that the reason for missing pixels in the area 203 is“poor image quality”. Also, regarding an area for which the reason formissing pixels has not been determined to be “poor image quality” (e.g.,the area 204), the composition unit 104 can estimate that the reason formissing pixels therein is the “shortage of images”.

Furthermore, the composition unit 104 can determine whether the“shortage of images” has occurred due to the “shortage of overlap” inused images, or has occurred due to the “failure in shooting”, which isa shortage of pieces of shot image data that are used to generate thecomposite image 201. The composition unit 104 determines whether the“shortage of images” has been caused by the “shortage of overlap” or the“failure in shooting” using the following method.

First, the composition unit 104 estimates the reason for missing pixelsin the area 202 using the estimation method that is used in estimatingthe reason for missing pixels in the area 204 (missing section C)(“shortage of images”). Consequently, the composition unit 104 estimatesthat the reason for missing pixels in each of the area 202 and the area204 is the “shortage of images”. Furthermore, the composition unit 104determines whether the reason for missing pixels in each area is the“failure in shooting” or “shortage of overlap” based on whether thenumber of missing pixels (the size of the missing pixel area) in each ofthe area 202 and the area 204 exceeds a predetermined threshold.

The predetermined threshold is the number of missing pixels, and isassumed to be four, for example. In a case where the composition unit104 determines that the number of mixing pixels in the area 204 of FIG.2 (one) does not exceed the predetermined threshold (e.g., four), itestimates that the reason for missing pixels in the area 204 is the“failure in shooting”. On the other hand, in a case where thecomposition unit 104 determines that the number of mixing pixels in thearea 202 (eight) exceeds the predetermined threshold (e.g., four), itestimates that the reason for missing pixels in the area 202 is the“shortage of overlap”.

Note that FIG. 2 depicts a case where the number of mixing pixels in thearea 202 (eight) is larger than the number of mixing pixels in the area204 (one). However, in a case where the shooting unit 101 has failed inshooting of a subject (there is a shortage of images that can be usedfor the composite image 201), there is a high possibility of theoccurrence of missing pixels throughout a wide range in the compositeimage 201). That is to say, as the missing pixel area attributed to thefailure in shooting (the area 204) is larger than the missing pixel areaattributed to the shortage of overlap (the area 202), the number ofmixing pixels in the area 204 can be larger than that in the area 202.Therefore, the composition unit 104 can determine whether the reason formissing pixels (“shortage of images”) is the “failure in shooting” or“shortage of overlap” by setting the predetermined threshold inconsideration of the number of mixing pixels.

Below is a description of another estimation method for a case where thereason for missing pixels is the “failure in shooting”. The compositionunit 104 obtains a communication history related to a defect in thecontact point between the image capturing apparatus (camera) and thelens in the shooting unit 101, which can cause the “failure in shooting”(information including time and an event, and information of a shutterinterval). Then, the composition unit 104 specifies used images thatwere shot at shooting times adjacent to the time of failure to releasethe shutter of the camera (the occurrence of a shooting failure event).In this way, the composition unit 104 can estimate that the reason formissing pixels in the area 204 is the “failure in shooting” based onwhether there is a missing pixel area in the surrounding of the usedimages that were specified from the composite image 201 using theforegoing method. Furthermore, a description is now given of anotherestimation method for a case where the reason for missing pixels is the“shortage of overlap”. The composition unit 104 may obtain the time atwhich a subject was shot when the moving speed of the mobile object(drone) on which the shooting unit 101 is mounted exceeded a threshold,and specify used images that were shot at shooting times adjacent to theobtained time. In this way, the composition unit 104 can estimate thatthe reason for missing pixels in the area 202 is the “shortage ofoverlap” based on whether there is a missing pixel area in thesurrounding of the used images that were specified from the compositeimage 201 using the foregoing method. The threshold for the moving speedof the mobile object (drone) may be any numerical value of a speed atwhich the “shortage of overlap” occurs when the shooting unit 101 shootsa subj ect.

After estimating the reasons for missing pixels in the areas 202 to 204,the composition unit 104 records respective estimation results in thecomposite image data in association with the coordinates of the areas202 to 204 in the composite image 201. The composition unit 104transmits the generated composite image data to the notification unit105.

Based on the determination information on the image qualities of piecesof shot image data from the determination unit 103 and on the compositeimage data from the composition unit 104, the notification unit 105provides a notification about methods of re-shooting a subject(inspection target surfaces) to be used by the shooting unit 101. In thecomposite image data, the coordinates of the areas 202 to 204 in thecomposite image 201 and the reasons for missing pixels in the areas 202to 204 are recorded in association with each other. The notificationunit 105 refers to the composite image data for the reasons for missingpixels in the areas 202 to 204, and presents re-shooting methodscorresponding to the reasons for missing pixels in association with thecoordinates of the areas 202 to 204 in the composite image 201.

FIG. 3 is a diagram showing an example of presentation of re-shootingmethods to be used by the information processing apparatus. Thenotification unit 105 provides a notification about the composite image201 and the re-shooting methods for the areas 202 to 204 shown in FIG. 3to respective units and the CPU (not shown) of the informationprocessing apparatus, as well as a terminal and the like (not shown) ofa user. The terminal of the user includes, for example, a smartphone, atablet, a notebook PC, and so forth.

In a case where the reason for missing pixels in the area 202 is the“shortage of overlap”, the notification unit 105 provides a notificationso that the speed at which the shooting range is changed when theshooting unit 101 shoots a subject (inspection target surfaces) isre-set. For example, in a case where the shooting unit 101 is mounted ona mobile object (drone), the notification unit 105 provides anotification so that the moving speed of the mobile object is reduced.If the speed at which the shooting range is changed with respect to thesubject (inspection target surfaces) is reduced, more pieces of shotimage data will overlap, and thus the “shortage of overlap” in thecomposite image 201 will be resolved.

In a case where the mobile object (drone) on which the shooting unit 101is mounted moves autonomously, the notification unit 105 may directlyaccess the shooting unit 101 and change the settings so as to reduce themoving speed of the mobile object. In a case where the mobile object(drone) on which the shooting unit 101 is mounted moves in accordancewith a user operation, the notification unit 105 may provide anotification to the terminal (not shown) and the like of the user sothat the user reduces the speed of the mobile object. Also, there arecases where the composition unit 104 generates a composite image basedon a group of still images extracted from moving images of the subject(inspection target surfaces) shot by the shooting unit 101. At thistime, in a case where the composite image 201 includes the area 202attributed to the “shortage of overlap”, the notification unit 105provides a notification so that the interval at which the still imagesare extracted (captured) from the moving images is reduced.

In a case where the reason for missing pixels in the area 204 is the“failure in shooting”, the notification unit 105 provides a notificationso that the interval at which the shutter of the shooting unit 101 isreleased is changed. For example, in a case where the shutter includedin the shooting unit 101 is controlled by a user operation, thenotification unit 105 provides a notification to the terminal and thelike of the user so that the interval of a shutter operation is reduced.In a case where the shutter included in the shooting unit 101 is aninterval shutter that is automatically operated, the notification unit105 provides a notification about re-setting of a shooting interval.Here, as the interval shutter is set so as to “shoot one still imageevery X seconds”, the notification is provided so as to reduce “Xseconds”. The notification unit 105 may directly access the shootingunit 101 and change the settings of the interval shutter. Also, in acase where the shutter included in the shooting unit 101 is controlledby a user operation, the notification unit 105 may provide anotification to the user so that a shutter operation is performedflawlessly.

In a case where the reason for missing pixels in the area 203 is “poorimage quality”, the notification unit 105 provides a notification sothat the shooting settings of the shooting unit 101 are changed. Inorder to change the shooting settings of the shooting unit 101, thenotification unit 105 uses determination information on the imagequalities of unused pieces of shot image data that have not beenreceived from the determination unit 103. In a case where the reason formissing pixels in the area 203 is “poor image quality”, thedetermination information always includes shot image data with imagequality information for which the image quality has not been determinedto be favorable. Image quality information for which the image qualityhas not been determined to be favorable is information indicating thatone of the shooting resolution, focus, and blurring is not favorable.The notification unit 105 decides on the content of change in theshooting settings in the shooting unit 101 with reference to imagequality information indicating unfavorable image quality.

In a case where it is determined, for example, that the “shootingresolution” is not favorable based on the determination information(determination result) on the image quality, the notification unit 105provides a notification so that the setting of the focal length of theshooting unit 101 is changed to the telephoto side. Once the setting ofthe focal length has been changed to the telephoto side, the shootingrange is reduced, and thus the notification unit 105 provides anotification so that the speed at which the shooting range of theshooting unit 101 is changed is reduced, or the shooting interval of theshooting unit 101 is reduced. Alternatively, the notification unit 105may provide a notification so that the shooting position of the shootingunit 101 further approaches the subject (inspection target surfaces)while maintaining the focal length of the shooting unit 101 as is. Inthis way, the shooting distance from the shooting unit 101 to thesubject (inspection target surfaces) is reduced, and thus the shootingresolution can be further increased.

In a case where it is determined that “focus” is not favorable based onthe determination information (determination result) on the imagequality, the notification unit 105 provides a notification so that thef-number of the shooting unit 101 is increased. As a result, the depthof field becomes deeper, thereby making it easier to bring the subject(inspection target surfaces) into focus.

In a case where it is determined that “blurring” is not favorable basedon the determination information (determination result) on the imagequality, the notification unit 105 provides a notification so that theshutter speed of the shooting unit 101 is increased. At this time,together with the foregoing notification, the notification unit 105 mayprovide a notification so that the ISO film speed is increased. Also,“blurring” may occur in a case where the speed at which the shootingrange of the shooting unit 101 is changed is too fast. In view of this,the notification unit 105 may provide a notification so that the speedat which the shooting range of the shooting unit 101 is changed isreduced. In a case where the mobile object (drone) on which the shootingunit 101 is mounted moves autonomously, the notification unit 105 maydirectly access the shooting unit 101 and set the moving speed of themobile object to be slower. On the other hand, in a case where themobile object on which the shooting unit 101 is mounted moves inaccordance with a user operation, the notification unit 105 may providea notification to the terminal (not shown) of the user so that themoving speed of the mobile object is reduced. At this time, in a casewhere the notification unit 105 notifies the terminal of the user of achange in the shooting settings of the shooting unit 101, it maydirectly access the shooting unit 101 and change the shooting settingsin place of the user.

FIG. 4 is a flowchart illustrating a flow of processing executed by theinformation processing apparatus. The following describes the flowchartof FIG. 4 with reference to FIG. 1A to FIG. 3 . Note that program codesthat are in conformity with the flowchart are stored in a memory of thestorage unit 102 (RAM, ROM, etc.) of the information processingapparatus 100. The present processing is realized by the CPU (not shown)and the like reading out and executing a program. Note that in thepresent flowchart, processing related to transmission and reception ofdata and the like may be executed either directly or via, for example, awired or wireless network.

In step S401, the shooting unit 101 shoots a subject of a structure tobe inspected (inspection target surfaces), and obtains pieces of shotimage data. The shooting unit 101 transmits the pieces of shot imagedata to the storage unit 102, and processing proceeds to step S402.

In step S402, the storage unit 102 stores the pieces of shot image data.The storage unit 102 transmits the pieces of shot image data to thedetermination unit 103, and processing proceeds to step S403.

In step S403, based on respective pieces of image quality information ofthe plurality of pieces of shot image data obtained by shooting thesubject, the determination unit 103 determines the image quality of eachpiece of shot image data. The determination unit 103 transmitsdetermination information (determination result) obtained by determiningthe image quality of each piece of shot image data to the storage unit102 and the notification unit 105, and processing proceeds to step S404.

In step S404, based on the determination information (determinationresult) of each piece of shot image data, the storage unit 102 selectspieces of shot image data for generating a composite image. That is tosay, the storage unit 102 selects, from among respective pieces of shotimage data, only pieces of shot image data for which the determinationinformation (determination result) on the image qualities indicatesfavorable image qualities. The storage unit 102 transmits the selectedpieces of shot image data to the composition unit 104, and processingproceeds to step S405.

In step S405, the composition unit 104 generates a composite image 201by compositing together the pieces of shot image data with favorableimage qualities using a known method (e.g., a method of panoramiccomposition or a method of three-dimensional reconfiguration), andprocessing proceeds to step S406.

In step S406, based on whether there is a missing pixel area (for whichno pixel value has been recorded) in the composite image 201, thecomposition unit 104 determines whether to change the shooting settingsof the shooting unit 101. In a case where the composition unit 104determines that there is a missing pixel area (for which no pixel valuehas been recorded) in the composite image 201 (Yes of step S406),processing proceeds to step S407. In a case where the composition unit104 determines that there is no missing pixel area (for which no pixelvalue has been recorded) in the composite image 201 (No of step S406),processing ends.

In step S407, the composition unit 104 determines areas 202 to 204 withmissing pixels (for which no pixel value has been recorded) in thecomposite image 201, and estimates the reasons for missing pixels in theareas 202 to 204. The composition unit 104 records the reasons formissing pixels into composite image data in association with respectivepositions (coordinates) of the areas 202 to 204. The composition unit104 transmits the composite image data to the notification unit 105. Thenotification unit 105 provides a notification about re-shooting methodsto be used by the shooting unit 101 based on the determinationinformation on the image qualities of the pieces of shot image data fromthe determination unit 103, and on the composite image 201 from thecomposition unit 104. Thereafter, processing proceeds to step S408.

In step S408, in accordance with a selection of a re-shooting method viaa user operation or the notification unit 105, the notification unit 105changes various types of settings related to re-shooting of the subject(inspection target surfaces) corresponding to the areas 202 to 204 to beperformed by the shooting unit 101, and processing returns to step S401.

As described above, according to the first embodiment, image qualitiesare determined based on pieces of image quality information of pieces ofshot image data, and a composite image is generated using images withfavorable image qualities based on determination information(determination result) on the image qualities. According to the firstembodiment, in a case where there is a missing pixel area in thecomposite image, a re-shooting method for the missing pixel area can beestimated from the relationship between shooting times of a used imageand an unused image. Consequently, in a case where there is a missingpixel area in the composite image, the missing pixel area can bevisualized as a location of re-shooting, and a re-shooting method forshooting a subject corresponding to the missing pixel area can bepresented.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. An information processing apparatus, comprising:a storage unit configured to store pieces of image quality informationrespectively for a plurality of images that show a subject; adetermination unit configured to determine whether image qualities ofthe images are favorable based on the pieces of image qualityinformation of the images; a composition unit configured to generate acomposite image using images that have been determined to have favorableimage qualities by the determination unit; and a notification unitconfigured to, in a case where there is a missing pixel area in thecomposite image, provide a notification about a re-shooting method forthe subject corresponding to the missing pixel area.
 2. The informationprocessing apparatus according to claim 1, further comprising estimationunit configured to, in a case where there is the missing pixel area inthe composite image, estimate a reason for missing pixels.
 3. Theinformation processing apparatus according to claim 2, wherein theestimation unit estimates a position of a used image in the compositeimage and estimates the reason for missing pixels based on whether thereis the missing pixel area in a surrounding of the used image, the usedimage having been used to generate the composite image and having ashooting time that is different, by a predetermined range or less, froma shooting time of an unused image that has not been used to generatethe composite image.
 4. The information processing apparatus accordingto claim 2, wherein the estimation unit estimates the reason for missingpixels based on whether there is a missing image between an imagelocated in a surrounding of the missing pixel area in the compositeimage and an image which is included among used images that have beenused to generate the composite image and which is adjacent to the imagein terms of a shooting order.
 5. The information processing apparatusaccording to claim 2, wherein the estimation unit estimates the reasonfor missing pixels based on whether a size of the missing pixel area inthe composite image exceeds a threshold.
 6. The information processingapparatus according to claim 2, wherein the estimation unit estimates,based on at least one of a time of occurrence of a shooting failureevent in shooting means for shooting the subject and a time at which thesubject is shot when a moving speed of a mobile object on which theshooting unit is mounted has exceeded a threshold, a position of a usedimage with a shooting time that is different from the time by apredetermined range or less in the composite image, and estimates thereason for missing pixels based on whether there is the missing pixelarea in a surrounding of the used image.
 7. The information processingapparatus according to claim 2, wherein the notification unit providesthe notification about the re-shooting method for the subjectcorresponding to the missing pixel area based on the reason for missingpixels estimated by the estimation unit.
 8. The information processingapparatus according to claim 2, wherein the reason for missing pixelsincludes at least one of a shortage of overlap, a failure in shooting,and poor image quality.
 9. The information processing apparatusaccording to claim 1, wherein the notification unit provides thenotification about the re-shooting method for the subject correspondingto the missing pixel area based on pieces of image quality informationof unused images that have not been used to generate the compositeimage.
 10. The information processing apparatus according to claim 1,wherein the re-shooting method for the subject corresponding to themissing pixel area includes a method of changing at least one of a speedat which a shooting range is changed at a time of shooting of thesubject, and a shooting setting at a time of shooting of the subject.11. The information processing apparatus according to claim 1, whereinthe notification unit notifies a terminal of a user of information inwhich a position of the missing pixel area in the composite image andthe re-shooting method for the subject corresponding to the missingpixel area are associated with each other.
 12. The informationprocessing apparatus according to claim 1, wherein the pieces of imagequality information of the images include a plurality of elements withwhich the image qualities of the images are evaluated, and thedetermination unit determines whether the image qualities of the imagesare favorable based on at least one of the plurality of elements. 13.The information processing apparatus according to claim 1, wherein thedetermination unit determines whether the image qualities of the imagesare favorable based on at least one of a shooting resolution, focus, andblurring.
 14. The information processing apparatus according to claim 1,wherein the storage unit stores a determination result of thedetermination made by the determination unit, and transmits, to thecomposition unit, images for which the determination result indicatesthat the image qualities are favorable.
 15. The information processingapparatus according to claim 1, further comprising a shooting unitconfigured to shoot the subject.
 16. The information processingapparatus according to claim 1, wherein the subject is a structure, andthe information processing apparatus is used in an inspection of thestructure.
 17. A shooting system, comprising: a mobile object includingshooting unit configured to shoot a subject; and an informationprocessing apparatus that processes images shot by the shooting unit,wherein the information processing apparatus includes a storage unitconfigured to store pieces of image quality information respectively fora plurality of images that show the subj ect, a determination unitconfigured to determine whether image qualities of the images arefavorable based on the pieces of image quality information of theimages, a composition unit configured to generate a composite imageusing images that have been determined to have favorable image qualitiesby the determination unit, and a notification unit configured to, in acase where there is a missing pixel area in the composite image, providea notification about a re-shooting method for the subject correspondingto the missing pixel area.
 18. A shooting system, comprising: a tripodhead; a shooting unit configured to shoot a subject, the shooting unitbeing mounted on the tripod head; and an information processingapparatus that processes images shot by the shooting unit, wherein theinformation processing apparatus includes a storage unit configured tostore pieces of image quality information respectively for a pluralityof images that show the subj ect, a determination configured todetermine whether image qualities of the images are favorable based onthe pieces of image quality information of the images, a compositionunit configured to generate a composite image using images that havebeen determined to have favorable image qualities by the determinationunit, and a notification unit configured to, in a case where there is amissing pixel area in the composite image, provides a notification abouta re-shooting method for the subject corresponding to the missing pixelarea.
 19. A method comprising: storing pieces of image qualityinformation respectively for a plurality of images that show a subject;determining whether image qualities of the images are favorable based onthe pieces of image quality information of the images; generating acomposite image using images that have been determined to have favorableimage qualities by the determining; and in a case where there is amissing pixel area in the composite image, providing a notificationabout a re-shooting method for the subject corresponding to the missingpixel area.
 20. A non-transitory computer-readable storage mediumstoring a program that, when executed by a computer, causes the computerto perform a method of an information processing apparatus comprising:storing pieces of image quality information respectively for a pluralityof images that show a subject; determining whether image qualities ofthe images are favorable based on the pieces of image qualityinformation of the images; generating a composite image using imagesthat have been determined to have favorable image qualities by thedetermining; and in a case where there is a missing pixel area in thecomposite image, providing a notification about a re-shooting method forthe subject corresponding to the missing pixel area.